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ABSTRACT 


The  United  States  Navy  has  established  a  Program  Office  for  Acquisition,  PMA- 
268,  to  develop  the  Navy  Unmanned  Combat  Air  System  (NUCAS).  The  NUCAS  will  be 
a  fighter-sized  aircraft  capable  of  a  variety  of  missions  including  deep-strike.  Intelligence 
Surveillance  and  Reconnaissance  (ISR),  Time  Sensitive  Targeting  (TST)  and  Air-to-Air 
Refueling  (AAR). 

The  NUCAS  will  offer  new  capabilities  to  the  operability  of  a  Carrier  Air  Wing 
(CAW).  Potential  benefits  include  improvements  in  combat  sortie  completion  rate  for 
manned  aircraft  such  as  the  F/A-18  Super  Hornet  and  the  F-35C  Lighting  II  Joint  Strike 
Fighter  (JSF). 

In  this  thesis,  we  evaluate  a  strike  scenario  that  focuses  on  the  coordination  of  the 
NUCAS,  the  F/A-18  Super  Hornet,  and  the  F-35C  Lightning  11.  We  construct  a 
simulation  model  of  the  scenario,  and  use  a  designed  experiment  to  run  12,000  simulated 
coordinated  strike  events.  We  then  use  a  variety  of  statistical  and  graphical  tools  to 
evaluate  the  result  in  order  to  determine  the  quantity  of  aircraft  required  for  mission 
success,  and  operational  factors  necessary  to  limit  friendly  aircraft  losses. 

The  results  indicate  that  a  division  of  four  NUCAS  aircraft  is  advantageous,  in 
terms  of  achieved  high  target  casualty  rates  and  high  blue  survivability  rates.  The  results 
also  highlight  the  necessity  of  stealth  technology  requirements  in  future  aircraft 
development. 
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THESIS  DISCLAIMER 


The  reader  is  cautioned  that  the  computer  programs  presented  in  this  research  may 
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the  time  available,  to  ensure  that  the  programs  are  free  of  computational  and  logical 
errors,  they  cannot  be  considered  validated.  Any  application  of  these  programs  without 
additional  verification  is  at  the  risk  of  the  user. 
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EXECUTIVE  SUMMARY 


The  Navy  Unmanned  Combat  Air  System  (NUCAS)  will  be  a  fighter-sized 
aireraft  eapable  of  a  variety  of  missions.  The  NUCAS  will  offer  signifieant  additions  to 
the  operability  of  a  Carrier  Air  Wing  (CAW)  with  improvements  in  eombat  effeetiveness 
and  airerew  survivability.  With  a  projected  combat  radius  of  1,500  to  2,100  nautical 
miles,  the  NUCAS  should  be  able  to  strike  targets  deep  within  enemy  territory.  When 
deployed  as  part  of  a  CAW,  the  NUCAS  must  safely  and  effectively  be  integrated  with 
the  over  70  other  jet  and  rotary-wing  aircraft  currently  operating  on  a  modern  naval 
aircraft  carrier. 

In  this  thesis,  we  examine  two  questions: 

•  What  combination  of  manned  and  unmanned  aircraft  provides  the  best 
mission  success  rate? 

•  What  operational  factors  allow  the  NUCAS  to  complete  its  mission  with 
limited  loss  rate  to  manned  and  unmanned  aircraft? 

We  design  a  coordinated  strike  scenario  using  an  agent  based  combat-modeling 
program  called  Map  Aware  Non-uniform  Automata  (MANA)  to  test  the  interaction  of  the 
NUCAS  among  current  and  future  manned  aircraft  such  as  the  F/A-18  Super  Hornet  and 
the  F-35C  Lightening  II  Joint  Strike  Fighter  (JSF).  This  coordinated  strike  scenario  is 
created  to  stress  the  capabilities  of  each  aircraft  in  order  to  gain  some  insight  into  the 
factors  that  provide  for  mission  success  and  limited  friendly  aircraft  losses.  A  variety  of 
unit  factors  are  included  in  the  simulation,  including  aircraft  starting  locations,  quantities, 
stealth,  various  sensor  characteristics,  and  various  weapons  characteristics.  By  utilizing 
design  of  experiment  techniques,  we  adjust  these  unit  factors  for  each  of  the  friendly  and 
enemy  agents,  thus  creating  240  different  numbers  of  combinations  that  we  evaluate  and 
analyze. 

Our  analysis  of  the  12,000  simulated  results  provides  some  insights  about  mission 
success  and  friendly  survivability: 


XIX 


•  NUCAS  aircraft  should  be  operated  in  a  group  of  at  least  three  to  four 
aireraft. 

•  F-35C  aircraft  should  operate  in  a  group  of  at  least  four  aircraft  when  used 
for  OCA  missions  against  a  large  number  of  enemy  aireraft. 

•  Stealth  is  an  important  factor  for  both  successful  target  destruetion  and 
blue  survivability. 

•  Weapons  Phit  is  a  major  factor  in  target  destruction  and  blue  survivability. 

Our  approach  shows  how  simulation,  data  farming  techniques,  and  data  analysis  can  be 
used  to  explore  CAW  operations.  The  results  of  our  analysis  can  be  used  to  assist  the 
U.S.  Navy  and  Naval  Air  System  Command  (NAVAIR)  as  they  continue  to  develop 
taetics  for  the  future  of  CAW  and  the  NUCAS.  By  modifying  the  simulation  model, 
adding  exploring  models  of  additional  missions,  or  using  this  approach  on  other  models 
of  NUCAS  eapabilities  and  CAW  operations,  additional  insights  to  the  eapabilities  of  the 
NUCAS  ean  be  addressed  as  continued  development  of  the  NUCAS  program  continues. 
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I.  INTRODUCTION 


Aircraft  carriers  are  one  of  Ameriea’s  key  power-projection  systems.  To 
ensure  their  continued  operational  effectiveness  and  survivability  in  the 
future  security  environment,  they  need  to  be  equipped  with  new  air 
platforms  with  greater  range,  greater  persistence,  and  improved  stealth. 
(Ehrhard  and  Work,  2007) 

A,  OVERVIEW 

Naval  aviation  has  changed  dramatically  over  the  past  deeade.  The  eatalyst  of  this 
ehange  was  the  collapse  of  the  Soviet  Union  and  the  rise  of  terrorist  eells  eapable  of 
eoordinating  the  horrifie  attaeks  of  September  11,  2001.  There  has  been  a  shift  from 
larger,  long-ranged,  single-role  aireraft  such  as  the  F-14B  Tomcat  and  A-6  Intruder,  to 
the  smaller,  shorter-range,  multirole  F/A-18C  Hornet  and  the  F/A-18E/F  Super  Hornet. 
The  F/A-18  has  been  the  “tip-of-the-spear”  aireraft  sinee  1983  and  has  proven  itself  to  be 
very  capable  in  carrier  and  combat  operations.  In  Operation  Desert  Storm,  a  seetion  of 
F/A-18  pilots  were  eredited  with  two  MiG-21  kills  followed  by  the  suecessful  bombing 
of  ground  targets  on  the  same  mission.  Ever  since  its  introduetion,  the  F/A-18  has 
continued  to  break  reeords  for  availability,  reliability,  and  maintainability  (Federation  of 
American  Scientists,  2010).  Within  the  next  few  years,  the  F/A-18  will  replaee  all  but  the 
E-2C  Hawkeye  Airborne  Early  Warning  (AEW)  aircraft,  and  the  MH-60  Seahawk 
helicopters  in  a  Carrier  Air  Wing  (CAW) 

A  future  aircraft,  the  F-35C  Fightning  II  Joint  Strike  Fighter  (JSF),  is  eurrently  in 
testing  and  is  planned  to  replace  the  older  F/A-18C.  The  F-35C  brings  a  decade  of 
advances  in  sensors  and  avionies,  vastly  improved  stealth  utilizing  internal  armaments, 
and  the  promise  of  improved  maintenance  and  reliability. 

As  remarkable  as  these  advances  are,  both  the  F/A-18  and  the  F-35C  are 
surprisingly  limited  in  their  eombat  radius  and  endurance.  The  typical  carrier  air  wing  of 
the  “Reagan  Years”  consisted  of  F-14Bs,  A-6s,  and  other  aireraft  having  combat  radii 
well  over  800  nautical  miles.  In  contrast,  our  current  F/A-18  aircraft  is  limited  to  roughly 
400  nautieal  miles  using  a  typieal  combat  configuration  of  air  and  ground  attack 
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weapons.  It  is  expected  that  the  future’s  F-35C  will  only  extend  combat  ranges  by 
another  100  nautical  miles  (Jane's  All  the  World's  Aircraft,  2009). 

An  example  of  how  combat  range  affects  the  Navy’s  ability  to  support  coalition 
forces  can  be  seen  in  air  combat  operations  of  Operation  Enduring  Freedom  (OEF).  U.S. 
F/A-18s  launch  from  the  aircraft  carrier  stationed  in  the  North  Arabian  Sea.  They  fly 
north,  along  the  nearly  400-mile  airspace  corridor  over  Pakistan,  before  reaching 
Southern  Afghanistan  where  they  proceed  to  their  first  tanker.  After  tanking,  they  can 
expect  to  report  on  station  and  support  ground  troops  for  roughly  two  hours  before 
requiring  their  second  tanking.  The  F/A-18  aircrew  will  then  spend  roughly  two  more 
hours  on  station  before  having  to  join  a  third  tanker  in  order  to  make  the  transit  south  and 
finally  recover  on  the  aircraft  carrier.  Tanking  operations  take  time  and  require  the 
aircraft  to  leave  its  area  of  responsibility  to  proceed  to  the  tanker  staging  area.  This 
greatly  reduced  the  persistence,  i.e.,  the  ability  of  an  aircraft  to  loiter  over  the  area  of 
concern.  There  is  clearly  room  for  improvement. 

B,  BACKGROUND  AND  MOTIVATION 

The  Navy  has  committed  to  developing  and  acquiring  the  Navy  Unmanned 
Combat  Air  System  (NUCAS).  The  NUCAS  will  be  a  fighter-sized  aircraft  capable  of  a 
variety  of  missions,  including  Deep-strike,  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR),  Time- Sensitive  Targeting  (TST),  and  Air  Refueling  (AF). 

The  NUCAS  will  significantly  advance  carrier  aviation  where  manned  and 
unmanned  aircraft  will  work  in  partnership  within  a  single  environment.  The  Navy 
anticipates  that  the  NUCAS  will  offer  significant  additions  to  the  operability  of  a  Carrier 
Air  Wing  with  improvements  in  combat  effectiveness  and  aircrew  survivability. 

The  NUCAS  has  the  potential  to  restore  the  U.S.  Navy’s  deep  strike  capabilities, 
and  to  extend  the  combat  radius  required  for  effective  combat  operations.  With  a 
projected  combat  radius  of  1,500  to  2,100  nautical  miles,  the  NUCAS  should  be  able  to 
strike  targets  deep  within  enemy  territory.  The  NUCAS  is  also  being  developed  with  the 
capability  of  receiving  in-flight  aerial  refueling,  thus  extending  the  NUCAS ’s  range 
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almost  indefinitely.  With  in-flight  refueling,  a  NUCAS  eould  eoneeivably  stay  airborne 
for  hundreds  of  hours  without  any  of  the  coneerns  that  have  affeeted  manned  aireraft, 
sueh  as  pilot  fatigue  or  performanee-related  blaekouts.  This  not  only  benefits  the  Navy’s 
deep-strike  eapability  but  also  inereases  the  NUCAS ’s  potential  loiter  time.  This  suggests 
that  ISR  missions  eould  greatly  benefit  from  the  nearly  unlimited  on-station  time  the 
NUCAS  provides. 

The  eurrent  NUCAS  Demonstrator,  Northrop  Grumman’ s  X-47B,  is  still  in  its 
development  phase.  The  eurrent  objeetives  (Northrop  Grumman,  2010)  are  to: 
demonstrate  the  technieal  feasibility  of  earner  landings  with  a  tail-less,  low  observable 
relevant  platform  prototype,  eontinue  the  maturation  of  relevant  earner  landing  and 
integration  teehnologies,  and  eonduet  aetual  NUCAS  earrier  landings. 

The  NUCAS  must  safely  and  effectively  be  integrated  with  the  over  70  other  jet 
and  rotary-wing  aircraft  currently  operating  within  the  cramped,  4.5  acres  of  a  modern 
naval  aircraft  canier.  Once  the  Navy  demonstrates  the  feasibility  of  NUCAS  carrier 
landings,  continued  development  will  determine  missions  and  tactics  that  effectively 
employ  the  NUCAS  within  the  CAW. 

2,  NUCAS  Missions 

The  two  most  obvious  missions  include  Intelligence  Surveillance  and 
Reconnaissance  (ISR)  and  Time  Sensitive  Targeting  (TST),  due  to  the  requirements  for 
extended  range  and  loiter  time  offered  by  the  NUCAS.  Air-to-ground  missions  for  the 
NUCAS  may  include  Close  Air  Support  (CAS),  long-range  strike,  and  coordinated  strike. 
Future  advances  may  include  an  air-to-air  capability  on  the  NUCAS  thus  expanding 
missions  to  include  Offensive  Counter  Air  (OCA),  and  Defensive  Counter  Air  (DCA). 

This  thesis  focuses  on  the  coordinated  strike  mission.  We  choose  the  coordinated 
strike  scenario  because  it  is  the  hallmark  training  scenario  for  all  fixed-wing  carrier 
aviators.  The  complexities  that  arise  from  operating  multi-purposed  aircraft  within  the 
same  battle-space  can  be  challenging.  Each  aircraft  has  its  area  of  responsibility  to 
include: 
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•  Initial  air-to-air  engagements 

•  Air-to-air  proteetion  for  the  air-to-ground  strike  aireraft 

•  Air-to-ground  strike  aireraft 

•  Eleetronie  Attaek  (EA)  aireraft 

•  Command,  Control  and  Communieations  (C3)  aireraft 

•  Signal  Intelligenee  (SIGINT)  aircraft 
In  this  thesis,  we  model  the  first  three  aspects. 

C.  RESEARCH  QUESTIONS 

Thus  the  objective  of  this  thesis  is  to  demonstrate  how  the  NUCAS  can  be 
evaluated  through  the  use  of  simulation,  design  of  experiments,  and  data  analysis. 
Specifically,  this  thesis  evaluates  a  coordinated  strike  scenario  utilizing  the  NUCAS  as  an 
air-to-ground  strike  aircraft  and  Offensive  Counter  Air  (OCA)  supplied  by  E/A-18  and  E- 
35C  aircraft.  Specific  questions  directing  this  thesis  include: 

•  What  combination  of  manned  and  unmanned  aircraft  provides  the  best 
mission  success  rate? 

•  What  operational  factors  allow  the  NUCAS  to  complete  its  mission  with 
limited  loss  rate  to  manned  and  unmanned  aircraft? 

D,  BENEFITS  OF  THE  STUDY 

The  Naval  Air  System  Command  (NAVAIR)  is  responsible  for  the  future 

capabilities  of  aircraft  and  aircraft-related  weapons  systems.  Effectively  integrating  the 

NUCAS  into  the  future  CAW  will  require  decisions  on  how  many  UCAVs  should  be  in 

an  airwing,  what  functions  they  should  be  expected  to  have,  how  their  presence  will 

change  current  airwing  operations,  etc.  The  findings  of  this  thesis  provides  preliminary 

insights  to  Navy  decision-makers  that  may  help  them  address  these  issues.  This  thesis 

also  serves  as  an  example  of  how  simulation  experiments  and  data  analysis  can  be 

leveraged  to  investigate  CAW  operations.  A  similar  approach  may  be  beneficial  for 
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assessing  a  variety  of  CAW  missions  involving  NUCAS.  As  the  NUCAS  moves  beyond 
its  demonstration  phase,  more  detailed  models  of  NUCAS  capabilities  and  CAW 
operations  can  be  developed. 

E.  METHODOLOGY 

Utilizing  analytical  techniques  developed  and  implemented  at  Naval  Postgraduate 
School  (NPS),  this  thesis  provides  some  preliminary  guidance  about  the  operational 
requirements  of  the  NUCAS  in  a  variety  of  mission  areas.  Design  of  experiments 
techniques  and  procedures  are  utilized  to  vary  NUCAS  operational  parameters.  We  create 
scenarios  using  an  agent-based  computer  simulation  modeling  platform  called  Map 
Aware  Non-uniform  Automata  (MANA).  By  using  MANA,  baseline  scenarios  can  be 
created  relatively  quickly,  and  then  investigated  using  an  approach  known  as  data 
farming.  In  data  farming,  simulation  parameters  are  varied  according  to  a  designed 
experiment,  to  create  a  set  of  well-chosen  variations  of  the  scenario.  These  variations  are 
simulated  multiple  times,  and  the  data  analyzed  to  identify  those  parameters  that  have  the 
greatest  effect  on  mission  success.  This  helps  future  developers  focus  on  the  parameters 
that  actually  make  the  biggest  impact,  therefore  providing  better  insights  about  NUCAS 
employment  and  effectiveness. 
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II.  SIMULATION  DEVELOPMENT 


A.  OVERVIEW 

This  chapter  provides  further  elaboration  on  the  NUCAS  and  the  simulation 
model  used  to  test  the  taeties  in  whieh  the  NUCAS  ean  be  expeeted  to  operate.  The 
purpose  of  this  thesis  is  not  to  reereate  every  eoneeivable  operational  environment  in 
whieh  the  NUCAS  might  be  employed,  but  to  look  at  a  eoordinated  strike  scenario 
simulation.  Our  seenario  is  eonstrueted  in  Map  Aware  Non-uniform  Automata  (MANA), 
an  agent-based  simulation  developed  by  the  Defense  Teehnology  Ageney  (DTA)  of  New 
Zealand. 

B,  MANA  SIMULATION 

The  primary  strength  of  MANA  is  that  it  can  be  utilized  effeetively  to  generate 
simulation  experiments  in  a  relatively  quick  fashion.  MANA  allows  various  assumptions 
and  agents  eharaeteristies  to  be  parametrieally  changed  and  the  simulation  run  again.  This 
faeilitates  the  ability  to  investigate  the  impaets  of  varying  several  independent  faetors  on 
mission  outeome.  MANA  is  well  doeumented,  providing  information  on  MANA’s  many 
uses,  agent  parameters,  and  capabilities. 

MANA  was  designed  by  DTA  to  researeh  the  often-random  eomplexities  of 
eombat  operations.  As  an  agent-based  simulation,  MANA  foeuses  on  individual  “agents” 
that  have  eharaeteristies  that  can  be  adjusted  utilizing  a  wide  variety  of  parameters.  Some 
of  the  major  agent  parameters  inelude: 

•  Type  (i.e.,  enemy,  friendly,  neutral) 

•  Quantity 

•  Speed 

•  Coneealment  or  stealth 

•  Weapons 
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Sensors 


•  Communications  between  agents 

•  Interaction  characteristics  between  various  other  agents 


MANA  is  extremely  customizable,  allowing  for  a  wide  variety  of  problems  to  be 
modeled  through  simulation.  Many  components,  such  as  sensors  and  weapons,  are  fairly 
detailed  and  include  parameters  such  as  probability  of  hit  and  probability  of  kill  for 
variable  ranges.  Other  aspects,  such  as  fuel  usage,  are  rather  simplistic,  with  only  a  single 
fuel-consumption  rate  for  a  particular  agent. 

DTA  performs  periodic  updates  to  MANA.  We  use  version  5.00,  known  as 
MANA  V,  for  this  thesis’s  simulation  building  process.  Figure  1  shows  the  start-up 
screen  for  MANA  V. 


MANA  V 


DTA** 


DEFENCE  TECHNOLOGY  AGENCY 


Vector-Based  Movement 

Map  Aware  Non-uniform  Automata 

This  software  has  been  developed  by  Gregory  McIntosh.  David  Galligan. 

Mark  Anderson  and  Michael  Lauren  for  the  New  Zealand  Army  and  Defence  Force. 

Terms  of  use; 

This  software  belongs  to  the  Defence  Technology  Agency  (DTA).  New  Zealand 
Defence  Force.  It  is  not  to  be  distributed  without  the  permission  of  DTA.  This 
product  is  not  to  be  offered  for  resale. 


See  hmjjWfcdf.-fnH.n^i  mana  for  latest  updates  and  support. 

I  understand  I  can  only  use  this  software  subject  to  the  conditions  stated  above. 
l/ei$ion  5.00.80 


I  Agree  ~|  |  Cancel~| 


pic:  J  Peel. 


Figure  1.  Screen  Shot  of  MANA  V  Startup  Screen  (From  Defense 
Technology  Agency,  2010) 
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C.  MISSION  SCENARIO 

A  Carrier  Strike  Group  (CSG)  is  operating  off  the  coast  of  a  threatening  nation. 
Intelligence  reports  that  the  threatening  nation  is  in  the  process  of  resurrecting  a  ground- 
based  anti-ship  missile  system  near  its  coast.  This  missile  system  has  been  deemed  a 
threat  to  region  stability. 

1.  Enemy  Forces 

Enemy  defenses  include  two  squadrons  of  Chinese-built  Chengdu  J-10  aircraft. 
All  enemy  aircraft  operate  from  a  known  airfield  located  on  the  coast.  Intelligence  reports 
that  the  enemy  aircraft  have  been  regularly  conducting  Combat  Air  Patrol  (CAP) 
missions  along  the  coast.  SIGINT  also  suggests  that  the  enemy  has  alert  aircraft  that  can 
be  launched  from  the  airfield  on  short  notice.  The  enemy  has  a  very  robust  early  warning 
detection  system,  providing  the  capability  to  classify  and  target  air  contacts  using  Ground 
Controlled  Interception  (CGI). 

2.  Friendly  Forces 

The  CAW  consists  of  three  squadrons  of  TACAIR  aircraft:  F/A-18F  Super 
Hornets,  F-35C  Fightning  IIs,  and  the  NUCAS  strike  aircraft.  The  F/A-18F  and  F-35C 
are  configured  for  air-to-air  engagements  and  are  primarily  responsible  for  protecting  the 
strike  aircraft.  The  NUCAS  is  the  strike  aircraft  and  is  configured  with  air-to-ground 
weapons.  The  NUCAS  is  the  strike  aircraft,  rather  than  a  manned  aircraft  because  of  the 
significant  threat  over  the  enemy  target  area. 

3.  Terrain  and  Scale 

The  Area  of  Operations  (AO A)  is  a  500  NM  by  500  NM  area  with  the  threatening 
country  towards  the  north  and  the  CSG  located  in  the  south.  The  enemy  J-10  CAP  aircraft 
are  already  on  station  and  will  respond  immediately  to  any  perceived  threat  that 
approaches  their  coast.  The  enemy  airfield  is  located  to  the  northeast  on  the  coast  and  will 
launch  aircraft  to  intercept  an  inbound  air  threat. 
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4. 


Mission 


The  Carrier  Air  Group  (CAG)  has  been  ordered  to  plan  and  conduct  a  coordinated 
air  strike  on  the  threatening  county.  The  primary  target  is  the  anti-ship  missile  system 
near  the  coast.  The  F/A-18F  and  F-35C  aircraft  will  provide  offensive  counter  air, 
targeting  any  enemy  aircraft  they  encounter.  The  NUCAS  will  strike  the  target  using 
GPS-guided  bombs.  We  also  assumed  that  EA,  C3,  and  SIGINT  capabilities  will  be 
provided  by  other  aircraft,  but  do  not  explicitly  model  these  in  the  simulation.  Figure  2 
shows  the  MANA  screen  shot  of  the  coordinated  strike  scenario. 
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Figure  2.  MANA  Screen  Shot  of  Coordinated  Strike  Scenario  (After 
Defense  Technology  Agency,  2010) 

D.  SIMULATION  MODELING  CHARACTERISTICS 

This  section  describes  the  various  characteristics  of  the  MANA  simulation  model 
that  we  use  for  this  thesis. 
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1. 


Goals  of  the  Simulation 


The  NUCAS  is  still  in  the  development  proeess  with  limited  test  flights  being 
scheduled  for  early  2010.  An  actual  NUCAS  carrier  landing  will  not  be  conducted  until 
late  2010  at  the  earliest.  It  is  expected  that  a  squadron  of  NUCAS  aircraft  will  not  be 
established  until  2015.  Using  an  actual  aircraft  in  a  war  game  against  an  enemy  threat 
simply  cannot  be  accomplished,  thus  the  reason  for  simulation. 

The  simulation  and  scenario  are  created  to  stress  the  capabilities  of  each  aircraft 
in  order  to  gain  some  insight  into  the  factors  that  provide  for  mission  success.  For  this 
simulation,  mission  success  is  measured  in  the  ability  of  the  friendly  agents  to 
successfully  destroy  the  target  area.  Friendly  losses  are  also  evaluated  because  losing  95 
percent  of  your  friendly  forces  to  destroy  a  target  is  not  considered  a  mission  success. 

In  order  to  determine  mission  success,  the  simulation  has  a  variety  of  unit  factors 
that  include:  location,  quantity,  stealth,  various  sensor  characteristics,  and  various 
weapons  characteristics.  By  utilizing  design  of  experiment  techniques,  we  adjust  these 
unit  factors  for  each  of  the  friendly  and  enemy  agents,  creating  an  abundant  number  of 
combinations  that  can  be  evaluated  and  analyzed. 

2,  Time,  Scale,  and  Map 

MANA  is  a  time-stepped  model  that  requires  pairing  between  the  simulation 
clock  and  the  actual  world  event  that  is  being  simulated.  This  simulation  uses  30  seconds 
to  represent  a  single  time  step.  The  simulation  time  typically  takes  no  more  than  235 
steps,  corresponding  to  just  less  than  two  hours  of  actual  time. 

MANA  refers  to  the  geographical  area  within  the  model  as  the  “battlefield”.  For 
this  simulation  the  battlefield  is  x  =  50.0  by  y  =  50.0  grid  which  corresponds  to  500  by 
500  nautical  miles  of  geographical  area.  Coordinates  start  in  the  upper  left  quadrant  of  the 
battlefield  and  increase  until  reaching  the  bottom  right  quadrant. 

MANA  has  an  upper  range  limit  of  100,000  meters,  or  62.14  miles.  MANA 

sensor  and  weapons  ranges  are  a  few  of  the  factors  that  have  this  limit.  Since  this  thesis 

required  sensor  ranges  beyond  62.14  miles  but  not  beyond  621.4  miles,  we  reduced  the 
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battlefield  seale  by  a  factor  of  10.  All  simulation  distances  are  reduced  by  this  factor  in 
order  to  overcome  the  MANA  limitation.  Figure  3  shows  the  battlefield  settings  menu, 
where  map  size  and  model  time  step  are  indicated. 

MANA  provides  a  means  for  maps  to  be  added  to  the  simulation  background.  A 
map  of  the  simulation  area  was  obtained  from  a  screen  capture  of  Google  Earth,  darkened 
using  Adobe  Photoshop,  and  then  saved  as  a  .bmp  image.  The  image  was  then  imported 
into  MANA  by  manually  updating  the  <Battlefield>  portion  of  the  code  in  the  extensible 
Markup  Language  (XML)  file.  Introducing  this  map  in  the  background  does  not  impact 
the  agents  being  simulated  or  their  behavior.  The  background  map  is  simply  used  to 
increase  user  situational  awareness  of  the  simulation  area. 


Ligure  3.  Screen  Shot  of  the  Battlefield  Setting  Menu  (Lrom  Defense 
Technology  Agency,  2010) 

3,  Enemy  Forces 

This  scenario  uses  three  enemy  agents  as  described  in  Table  1. 
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Agent 

Icon 

Description 

CAP  J- 10 

f 

Aircraft  already  in  flight  (on  patrol)  at  the 

start  of  simulation. 

Alert  J-10 

f 

Aircraft  on  alert  at  the  airfield  for  launch  as 

directed  (random). 

Target  Area 

+ 

The  ground-based  anti-ship  missile  site. 

Table  1 .  Enemy  Agents  Used  in  MANA  Simulation 


MANA  uses  the  term  “squads”  to  represent  agents.  The  seenario  starts  with  all 
squads  in  position;  their  start  loeations  are  determined  by  a  home  position  and  box  size 
centered  on  the  home  location.  The  CAP  J-lOs  are  located  at  their  CAP  station, 
designated  in  MANA  as  x  =  25  and  y  =  8.  The  individual  aircraft  associated  with  this 
squad  are  randomly  populated  within  a  rectangle  box  of  x  =  5.0  and  y  =  2.5.  As 
previously  mentioned,  all  measurements  are  reduced  by  a  factor  of  10;  therefore  this 
corresponds  to  a  real-world  area  of  50  by  25  nautical  miles. 

The  Alert  J-lOs  are  located  at  their  associated  airfield  located  in  MANA  as  x  =  40, 
y  =  12.  Once  the  simulation  starts,  a  countdown  timer  of  3600  seconds  (1  hour)  is 
activated.  All  Alert  J-lOs  will  launch  at  random  intervals  until  the  3600  seconds  is 
reached. 

In  MANA,  “Squad  Personalities”  determine  how  squads  react  with  other  agent 
squads.  Various  levels  of  personalities  can  be  adjusted  from  -100  to  +100,  thus  allowing 
for  a  wide  variety  of  squad  characteristics  and  behaviors.  For  example,  a  squad  assigned  a 
-100  to  “Enemy  Threat”  would  respond  by  moving  away  from  that  threat.  Conversely,  a 
squad  assigned  a  +100  would  move  towards  that  threat.  Both  the  CAP  J-lOs  and  Alert  J- 
10s  are  set  to  pursue  all  threat  aircraft  with  +100.  If  no  threat  aircraft  are  within  sensor 
range,  the  aircraft  have  a  secondary  personality,  of  +  15,  to  move  towards  their  waypoint 
located  at  x  =  25,  y  =  15. 
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Another  factor  of  a  squad  is  what  MANA  refers  to  as  “concealment.” 
Concealment  affects  the  probability  that  a  squad  can  be  seen  in  a  given  time  step.  For 
example,  a  squad  having  a  concealment  value  of  1.00  is  completely  invisible.  Conversely, 
a  squad  having  a  value  of  0.0  is  completely  visible.  For  aircraft  like  the  J-10,  this 
concealment  value  can  be  associated  with  stealth.  Both  the  J-10  squads  are  given  an 
adjusted  concealment  level  from  25  to  90  percent  of  the  base  concealment.  This 
corresponds  to  a  real-world  RADAR  Cross  Section  (RCS)  of  0.75  to  0.1  m^  RCS 
(Grining,  2000). 

Sensor  range  can  also  be  adjusted.  Both  J-10  sensor  classification  ranges  are  set  to 
the  maximum  limit  of  100,000  meters  (621.4  miles).  This  maximum  range  is  chosen  to 
simulate  the  added  advantage  the  enemy  country  has  when  classifying  air  targets  as 
threats.  We  assume  that  the  enemy  has  established  a  no-fly  exclusion  zone,  so  any  aircraft 
approaching  their  coast  will  be  declared  hostile  and  fired  upon. 

MANA  also  allows  for  a  wide  variety  of  weapons  to  be  employed  for  each  squad. 
The  J-10  employs  two  primary  air-to-air  missiles:  The  PL- 12,  medium-range,  active- 
radar  missile  and  the  PL-8,  short-range,  infrared  seeking  missile.  The  PL-12  and  PL-8 
quantity,  range,  and  probability  of  Kill  (Pk)  can  be  adjusted  in  MANA’s  “Weapons”  tab 
of  the  “Squad  Properties”  menu.  Using  design  of  experiments,  we  adjust  each  of  these 
factors  during  the  simulation.  Table  2  describes  the  factors,  along  with  their  minimum 
and  maximum  values. 

The  “Target  Area”  is  specified  as  a  squad  as  well.  However,  the  target  area  is  very 
different  from  the  other  squads  since  it  does  not  have  a  movement,  personalities, 
weapons,  or  sensors.  The  Target  Area  has  only  two  parameters  associated  with  it.  The 
first  is  a  threat  level  that  is  only  targeted  by  the  NUCAS.  The  second  parameter  is  a 
requirement  that  two  NUCAS  hits  are  necessary  in  order  for  the  target  to  be  destroyed. 
These  two  parameters  are  necessary  in  order  to  keep  air-to-air  units  from  attacking  the 
Target  Area,  and  to  create  a  more  realistic  scenario  in  which  two  bomb  hits  are  required 
for  successful  destruction. 
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Agent 

Factor 

Min 

Max 

Explanation  (Units) 

Notes 

J-10  CAP 

Number  of  Agents 

1 

20 

Number  of  aircraft 

Movement  Speed 

25 

80 

Speed  (knots  x  10) 

Personal  Concealment 

25 

90 

Probability  of  detection  (%) 

0.75—0.1  m2RCS 

PL-12  Weapon  Shots/ Ammo 

1 

6 

Number  of  weapons 

PL-12  Weapon  Range 

5632 

8046 

Weapons  range  (meters  x  10) 

35 — 50  miles 

PL-12  Weapon  Hit  Rate  per  Discharge 

0 

1.00 

Probability  of  hitting  target  (%) 

PL-8  Weapon  Shots/ Ammo 

2 

4 

Number  of  weapons 

PL-8  Weapon  Range 

322 

1609 

Weapons  range  (meters  x  10) 

2—1 0  miles 

PL-8  Weapon  Hit  Rate  per  Discharge 

0 

1.00 

Probability  of  hitting  target  (%) 

J- 10  Alert 

Number  of  Agents 

1 

20 

Number  of  aircraft 

Movement  Speed 

25 

80 

Speed  (knots  x  10) 

Personal  Concealment 

25 

90 

Probability  of  detection  (%) 

0.75— 0.1  m2  RCS 

Table  2.  Table  of  Enemy  Faetors  and  Their  Assoeiated  Min/Max  Values 

4,  Friendly  Forces 

The  seenario  uses  three  main  friendly  agents,  as  described  in  Table  3. 


Agent 

Icon 

Description 

NUCAS 

Unmanned  strike  aircraft  configured  with 

air-to-ground  bombs. 

F/A-18F 

OCA  aircraft  configured  with  air-to-air 

missiles. 

F-35C 

OCA  aircraft  configured  with  air-to-air 

missiles. 

Table  3.  Friendly  Agents  Flsed  in  MANA  Simulation 


The  friendly  squads  start  the  simulation  on  the  southern  portion  of  the  battlefield. 
The  NUCAS  are  located  the  furthest  south  at  the  start  position  of  x  =  25  and  y  =  47.5. 
Since  the  NFICAS  are  the  only  aircraft  in  the  strike  package  with  an  air-to-ground 
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weapon,  it  is  vital  that  other  aireraft  proteet  the  NUCAS  from  any  air-to-air  threats.  These 
protection  aircraft  are  referred  to  as  Offensive  Counter  Air  (OCA)  and  they  have  the 
responsibility  of  clearing  the  airspace  of  enemy  aircraft.  The  F-35C  and  F/A-18F  are  the 
OCA  and  will  have  a  start  location  of  x  =  25;  their  y  =  locations  varies  between  27.5  to 
47.5  placing  the  OCA  aircraft  in  front  of  the  strike  NUCAS. 

In  addition  to  the  start  location,  each  of  the  friendly  squads  also  has  waypoints 
which  they  move  towards.  The  first  waypoint  is  located  at  x  =  25  and  y  =  3.5  and 
corresponds  to  the  target  location.  Once  the  friendly  squads  reach  the  target  waypoint,  a 
second  waypoint  is  activated  and  a  return  to  the  final  location  at  x  =  25  and  y  =  47.5  is 
directed. 

The  squad  personalities  for  the  NUCAS  and  the  other  blue  aircraft  are  very 
different  from  each  other.  The  NUCAS  has  two  threat  levels  that  it  uses  to  differentiate 
between  the  J-10  threat  and  the  Target  Area  threat.  The  NUCAS  will  proceed  towards  the 
Target  Area  with  a  personality  weight  of +100  and  it  will  move  away  from  the  J-10  threat 
with  a  personality  weight  of  -30.  The  F/A-18  and  the  F-35C  have  only  one  threat,  the  J- 
10s,  and  they  will  move  towards  that  threat  with  a  personality  weight  of  +100.  Also,  in 
order  to  keep  all  of  the  blue  squads  moving  along  the  flight  path,  we  implemented  a 
personality  weight  of  +30  for  proceeding  to  the  specified  waypoint. 

The  MANA  concealment  factor  for  the  blue  squad  representation  is  the  same  as 
for  the  J-10,  except  the  values  are  higher  to  represent  the  smaller  RCS  values  of  the 
stealthy  F-35C  and  NUCAS.  Table  4  lists  the  personal  concealment  values  for  each  of  the 
friendly  aircraft  in  the  simulation  model. 

Research  done  on  the  sensor  classification  range  and  aperture  provided  a  wide 
variety  of  capabilities.  The  sensor  systems  for  all  of  the  blue  squads  are  classified,  but  we 
obtained  reasonable  numbers  from  unclassified  sources  such  as  Jane’s  Aircraft.  The 
NUCAS  sensor  systems  have  not  been  developed  at  the  time  of  this  thesis  so  assumptions 
are  necessary.  Since  the  NUCAS  is  smaller  than  the  F/A-18,  we  assume  that  the  sensor 
systems  for  the  NUCAS  will  have  to  be  small  as  well,  therefore  the  NUCAS  will  have  a 
range  and  aperture  arc  similar  to  those  of  a  current  F/A-18F  aircraft.  The  F-35  RADAR, 


16 


the  AN/APG-81,  will  have  substantial  higher  ranges  and  eapability  than  the  eurrent 
F/A-18’s  AN/APG-74  (Northrop  Grumman  2010).  Table  4  lists  sensor  classification 
ranges  and  aperture  arcs  for  each  of  the  blue  squads. 


Agent 

Factor 

Min 

Max 

Explanation  (Units) 

Notes 

NUCAS 

Number  of  Agents 

1 

12 

Number  of  aircraft 

Movement  Speed 

25 

65 

Speed  (knots  x  10) 

Personal  Concealment 

98 

100 

Probability  of  detection  (%) 

0.0015-0.0001  m2RCS 

Sensor  Class.  Range 

9656 

16093 

Sensor  range  (meters  x  10) 

60—100  miles 

Sensor  Aperture  Arc 

40 

120 

Sensor  width  (degrees) 

GBU-3 1  Weapon  Shots/Ammo 

1 

2 

Number  of  weapons 

GBU-3 1  Weapon  Range 

805 

2736 

Weapons  range  (meters  x  10) 

5—17  miles 

GBU-3 1  Weapon  Hit  Rate 

0 

1.00 

Probability  of  hitting  target  (%) 

F/A-18F 

Number  of  Agents 

1 

12 

Number  of  aircraft 

Home  Waypoint  Location  y: 

47.5 

27.5 

Distance  aircraft  is  in  front  of  NUCAS 

0-200  nm 

Personal  Concealment 

25 

90 

Probability  of  detection  (%) 

0.75-0.1  m2RCS 

Sensor  Class.  Range 

9656 

16093 

Sensor  range  (meters  x  10) 

60—100  miles 

Sensor  Aperture  Arc 

40 

120 

Sensor  width  (degrees) 

AIM- 120  Weapon  Shots/Ammo 

1 

6 

Number  of  weapons 

AIM-120  Weapon  Range 

4989 

15289 

Weapons  range  (meters  x  10) 

3 1—95  miles 

AIM- 120  Weapon  Hit  Rate 

0 

1.00 

Probability  of  hitting  target  (%) 

AIM-9  Weapon  Shots/ Ammo 

2 

4 

Number  of  weapons 

AIM-9  Weapon  Range 

966 

1770 

Weapons  range  (meters  x  10) 

6 — 11  miles 

AIM-9  Weapon  Hit  Rate 

0 

1.00 

Probability  of  hitting  target  (%) 

F-35C 

Number  of  Agents 

1 

12 

Number  of  aircraft 

Home  Waypoint  Location  y: 

47.5 

27.5 

Distance  aircraft  is  in  front  of  NUCAS 

0-200  nm 

Personal  Concealment 

98 

100 

Probability  of  detection  (%) 

0.005-0.0015  m2  RCS 

Sensor  Class.  Range 

9656 

19312 

Sensor  range  (meters  x  10) 

60—120  miles 

Sensor  Aperture  Arc 

40 

180 

Sensor  width  (degrees) 

Table  4.  Table  of  Friendly  Factors  and  Their  Associated  Min/Max  Values 
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The  friendly  squad  weapons  systems  are  very  similar  to  those  of  the  enemy 
squads.  The  F/A-18F  and  F-35C,  configured  for  air-to-air  combat,  will  carry  a 
combination  of  AIM-120  AMRAAM  and  AIM-9  Sidewinder  missiles.  The  AIM-120  is  a 
medium-range,  active-RADAR  missile.  The  AIM-9  is  a  short-range,  infrared-seeking 
missile.  The  AIM- 120  and  AIM-9  are  broken  down  into  just  three  factors  for  each 
weapon:  quantity,  range,  and  probability  of  hit. 

The  NUCAS  will  carry  two  internal  GBU-31  Joint  Direct  Attack  Munitions 
(JDAM).  The  JDAM  is  a  2000  lb  GPS  guided  bomb  used  for  ground  targets.  The 
NUCAS  will  not  have  air-to-air  weapons  capabilities.  The  weapons  factors  for  the  F/A- 
18F,  F-35C,  and  NUCAS  are  also  listed  in  Table  4. 

5,  Sources,  Abstractions,  and  Assumptions 

It  is  important  that  the  sources  of  input  data,  model  abstractions,  and  assumptions 
are  addressed.  Obtaining  the  source  material  for  effectively  simulating  the  aircraft  and 
weapons  systems  in  MANA  took  a  large  percentage  of  the  modeling  process  time.  The 
sources  for  the  performance  characteristics  were  obtained  from  a  wide  variety  of 
unclassified  sources,  including,  Jane’s  Aircraft,  the  Federation  of  American  Scientists, 
Global  Security.org,  and  an  assortment  of  aircraft  and  weapons  defense  corporation  Web 
sites. 

Air  combat  is  a  very  physics-based  environment  with  weight,  thrust,  drag,  and  lift 
being  common  characteristics.  Air  combat  is  also  a  three-dimensional  environment  where 
altitude  advantages  often  drive  the  tactics  that  are  developed.  Modem  day  weapons, 
including  missiles  and  self-steering  bombs,  also  benefit  from  being  studied  in  a  physics- 
based,  three-dimensional  world. 

MANA  is  very  limited  in  its  ability  to  model  altitude;  therefore,  this  thesis 
abstracts  all  squad  aircraft  and  weapons  characteristic  by  placing  them  at  the  same 
operational  altitude.  In  short,  the  simulation  has  abstracted  the  three-dimensional  world 
of  air-combat  into  a  top-down,  two-dimensional  simulation  model. 
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Sensor  and  weapons  ranges  are  also  very  simplified.  Eaeh  of  the  squads  in  this 
simulation  uses  a  single  elassifieation  range.  This  is  unlike  real-world  operations  where 
detection  probability  tends  to  decrease  with  range.  The  probability  of  detecting  a  small 
RCS  contact  decreases  as  the  contact  gets  further  from  to  the  detection  sensor.  Weapons 
ranges  and  probability  of  hit  (Phit)  are  also  simulated  in  this  fashion,  with  only  a  single 
range  and  Phit  for  each  of  the  weapons  parameters.  MANA  does  have  the  ability  to  assign 
ranges  with  various  Phit  for  those  ranges,  however,  that  feature  was  not  utilized  for  this 
thesis  since  the  additional  detail  was  not  necessary  given  the  simplification  of  altitude. 

Fuel  consumption  is  an  abstraction  in  MANA  offering  limited  application.  A 
single  value  of  fuel  consumption  per  time-step  can  be  assigned;  however,  combat  aircraft 
do  not  consume  fuel  at  a  constant  rate.  A  multitude  of  factors  such  as  speed,  altitude,  and 
configuration,  all  provide  various  factors  to  the  fuel  consumption  rate.  Since  the 
coordinated  strike  scenario  does  not  challenge  the  combat  radius  of  the  aircraft  being 
simulated,  it  was  decided  to  eliminate  the  effects  of  fuel  from  the  simulation. 

Communication  between  squads  is  another  characteristic  that  can  be  defined  in 
MANA.  This  simulation  assumes  that  all  communications  are  perfect  and  do  not  have  a 
limited  range. 

Failure  of  equipment  and  operator  error  is  a  characteristic  that  plagues  modern- 
day  aircraft  and  systems.  This  simulation  assumes  that  equipment  failure  does  not  exist; 
however,  A  Phit  range  from  0.0  to  1.00  is  defined  for  each  of  the  weapons  used  in  the 
simulation. 


6,  Summary 

“Essentially,  all  models  are  wrong,  but  some  are  useful”,  is  an  oft  quoted  phrase 
from  the  statistician  George  E.  P.  Box  (DeGroot,  1987).  The  intent  of  this  thesis  is  to 
evaluate  the  effectiveness  of  using  NUCAS  as  a  force-multiplier  in  future  Carrier  Air 
Wings.  Simulation  models  are  a  useful  way  to  mitigate  the  human  and  financial  risk 
involved  in  actual  air-to-air  combat. 
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III.  EXPERIMENTAL  DESIGN 


A.  INTRODUCTION 

Data  farming  is  an  approach  where  multiple  variations  of  a  model  are  generated 
by  varying  the  input  parameters  in  a  well-ehosen  manner,  using  experimental  design  (see 
Sanchez  and  Wan,  2009).  In  this  thesis,  we  utilize  data  farming  teehniques  to  ereate  a 
large  data  set  by  simulating  a  multitude  of  faetor  eombinations  over  a  substantial  number 
of  iterations.  Through  data  analysis,  analysts  ean  better  understand  how  these  faetors  (and 
interaetions  among  these  faetors)  affeet  the  measures  of  performanee  and  measures  of 
effeetiveness.  The  analysis  may  help  deeision-makers  make  the  kinds  of  assessments  that 
ean  ultimately  lead  to  real-world  suecess. 

B,  FACTORS  OF  INTEREST 

For  our  simulation  experiment,  we  focus  on  two  types  of  faetors:  eontrollable  and 
uneontrollable.  Controllable  factors  are  those  that  the  deeision  maker  has  control  of  in  the 
real  world,  either  through  development  or  employment.  These  faetors  might  inelude 
aireraft  speed,  weapons  quantity,  or  sensor  range.  Quite  often,  the  eontrollable  faetors  are 
also  ealled  the  deeision  variables  or  deeision  faetors.  Uneontrollable  faetors  are  those  that 
eannot  be  eontrolled  in  the  real  world,  even  though  they  ean  be  eontrolled  within  the 
simulation.  These  might  inelude  enemy  eharacteristics  sueh  as  quantity,  speed,  sensor 
range,  and  weapon  performanee,  or  might  inelude  environmental  eharacteristics  such  as 
wind  speed  and  eloud  eover.  Often  these  are  referred  to  as  the  noise  variables  or 
uneontrollable  variables.  The  uneontrollable  and  eontrollable  factors  in  this  thesis  are 
summarized  in  Chapter  II,  Table  2  and  4,  respeetfully.  We  now  diseuss  them  in  further 
detail. 

The  MANA  seenario  uses  nautieal  miles  for  battlefield  dimensions  and  movement 
speeds,  however,  parameters  like  sensor  range  and  weapons  range  use  metric  units.  In 
many  cases,  eonversions  are  neeessary  in  order  to  suoeessfully  implement  the  data  into 
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MANA.  The  most  common  conversion  is  meters  to  miles  (1  mile  =  1,609.344  meters) 
and  meters  to  nautical  miles  (1  nautical  mile  =  1852  meters). 

1.  Controllable  Factors 

The  following  controllable  factors  are  the  main  factors  in  the  experimental  design. 
For  consistency,  the  labels  are  the  same  as  those  produced  in  MANA  output,  and  the 
acronym  NUCAS  is  shortened  to  UCAS. 

a.  BlueSpd 

Blue  speed  is  the  speed,  in  knots,  of  the  friendly  aircraft.  Blue  speed  varies 
from  250  to  650  knots.  Since  the  scenario  requires  that  the  strike  package  maintain 
formation,  the  same  speed  is  used  for  all  friendly  aircraft  within  a  particular  simulation 
run.  This  technique,  known  as  lock-stepping,  reduces  the  number  of  distinct  factors  that 
are  varied  for  the  experimental  design.  As  discussed  in  Chapter  II,  Blue  speed  is 
converted  to  MANA  units  by  dividing  the  speed  (in  knots)  by  ten.  A  speed  of  500  knots 
corresponds  to  a  MANA  speed  of  50. 

b.  UCASQnty 

UCAS  quantity  is  the  number  of  UCAS  aircraft.  This  number  varies  from 
1  to  12.  The  upper  limit  of  12  is  chosen  because  current  CAW  squadron  aircraft  are 
limited  to  roughly  10  to  12  aircraft. 

c.  UCASStlth 

UCAS  stealth  is  the  UCAS  aircraft  concealment  in  percent.  This  value 
varies  from  98  to  100.  These  values  correspond  to  a  RCS  of  0.0015  to  0.0001  m^ 
(Grining,  2000).  We  used  a  conversion  of  100  -  {RCS  xlOOO)  =  concealment.  Integer 
values  are  required  for  MANA;  therefore,  concealment  is  rounded  to  the  nearest  integer 
value. 
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d.  UCASSnsrRnge 


UCAS  sensor  range  is  the  distanee  at  whieh  the  UCAS  elassifies  other 
squads.  This  value  varies  from  60  to  100  nautieal  miles.  These  distanees  are  based  on  a 
similar  radar  system  used  in  the  F/A-18,  the  AN/APG-73  (Jane's  Avionies,  2010). 

e.  UCASSnsrApture 

UCAS  sensor  aperture  is  the  sensor  are  width.  This  value  varies  from  40  to 
120  degrees  (Jane's  Avionics,  2010).  Squads  that  are  within  the  range  and  aperture  of  the 
UCAS  sensor  will  be  classified;  squads  that  are  outside  the  range  or  aperture  of  the 
UCAS  sensor  will  not. 

f.  UCASWpnQnty 

UCAS  weapons  quantity  is  the  number  of  GBU-31  JDAM  weapons  that 
each  UCAS  carries,  and  is  either  1  or  2.  The  GBU-31  is  a  2,000-pound  Mk84  bomb  fitted 
with  a  GPS-guidance  kit  (Wikipedia,  2010).  The  maximum  weapons  load-out  for  the 
UCAS  is  4,500  pounds  (Northrop  Grumman,  2010),  thus  the  reason  for  the  upper  limit. 

g.  UCASWpnRange 

GBU31  weapons  range  is  the  range  of  the  GBU-31  JDAM.  This  value 
varies  from  5  to  17  miles  (Wikipedia,  2010). 

h.  GBUSlWpnPhit 

GBU3 1  weapon  probability  of  hit  (Phit)  is  the  probability  that  the  GBU-3 1 
hits  the  target.  This  value  varies  from  0.0  to  1.00. 

L  FlSQnty 

F/A-18  quantity  is  the  number  of  F/A-18E/F  Super  Hornet  aircraft.  This 
number  varies  from  1  to  12.  The  upper  limit  of  12  is  chosen  because  current  CAW 
squadron  aircraft  are  limited  to  roughly  10  to  12  aircraft. 
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7- 


FlSLead 


F/A-18  lead  is  the  distance,  in  nautical  miles,  that  the  F/A-18E/F  Super 
Hornet  is  in  front  of  the  strike  aircraft.  This  lead  distance  is  important  in  that  it  gives  the 
F/A-18  time  to  detect  and  destroy  a  threat  before  the  threat  targets  the  strike  aircraft.  This 
distance  varies  from  0  to  200  nautical  miles.  F/A-18  lead  distance  is  converted  to  a  y- 
coordinate  start  location  (y  =  27.5  to  47.5)  when  inputted  into  the  MANA  model. 

k.  FlSStlth 

F/A-18  stealth  is  aircraft  concealment  in  percent.  This  value  varies  from 
25  to  90.  These  values  correspond  to  a  RCS  of  0.075  to  0.01  m^  (Grining,  2000).  A 
conversion  of  100  -  {RCS  x  1000)  =  concealment  is  used. 

I  FlSSnsrRnge 

F/A-18  sensor  range  is  the  distance  at  which  the  F/A-18E/F  Super  Hornet 
classifies  other  squads.  This  value  varies  from  60  to  100  nautical  miles.  These  distances 
are  based  on  the  AN/APG-73  RADAR  characteristics  (Jane's  Avionics,  2010). 

m,  FlSSnsrApture 

E/A-18  sensor  aperture  is  the  sensor  arc  width.  This  value  varies  from  40 
to  120  degrees.  Squads  that  are  within  the  range  and  aperture  of  the  E/A-18  sensor  will  be 
classified;  squads  that  are  outside  the  range  or  aperture  of  the  E/A-18  sensor  will  not. 

n,  AIMllOWpnQnty 

AIM- 120  weapons  quantity  is  the  number  of  AIM- 120  AMRAAM  missile 
carried  by  either  the  E/A-18  or  the  E-35C.  AIM- 120  weapons  quantity  varies  from  1  to  6. 
This  factor  is  lock-stepped,  ensuring  that  every  E/A-18  and  E-35Cs  will  carry  the  same 
quantity  of  AIM- 120  missiles. 
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o.  AIMllOWpnRange 


AIM- 120  weapons  range  is  the  range  of  the  AIM- 120  AMRAAM  missile 
earned  by  either  the  F/A-18  or  the  F-35C.  This  value  varies  from  31  to  95  nautieal  miles 
(Jane's  Air-Launehed  Weapons,  2010).  This  faetor  is  look-stepped  between  the  F/A-18s 
and  F-35C. 


p.  AIMllOWpnPhit 

AIM- 120  weapon  probability  of  hit  (Phit)  is  the  probability  that  the  AIM- 
120  AMRAAM  missile  hits  the  target.  This  value  varies  from  0.0  to  1.00. 

q.  AIM9WpnQnty 

AIM-9  weapons  quantity  is  the  number  of  AIM-9  Sidewinder  missile 
earned  by  either  the  F/A-18  or  the  F-35C.  AIM-9  weapons  quantity  varies  from  2  to  4. 
This  faetor  is  look-stepped,  therefore,  every  F/A-18  and  F-35C  oarries  the  same  quantity 
of  AIM-9  missiles. 


r.  AIM9WpnRange 

AIM-9  weapons  range  is  the  range  of  the  AIM-9  Sidewinder  missiles 
earned  by  either  the  F/A-18  or  the  F-35C.  This  value  varies  from  6  to  11  nautieal  miles 
(Jane's  Air-Launched  Weapons,  2009).  This  factor  is  lock-stepped  between  the  F/A-18s 
and  F-35C. 


s.  AIM9WpnPhit 

AIM- 120  weapon  probability  of  hit  (Phit)  is  the  probability  that  the  AIM- 
120  AMRAAM  missile  hits  the  target.  This  value  varies  from  0.0  to  1.00. 
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t  F35Qnty 


F-35  quantity  is  the  number  of  F-35C  JSF  aircraft.  This  number  varies 
from  1  to  12.  The  upper  limit  of  12  is  chosen  because  current  CAW  squadron  aircraft  are 
limited  to  roughly  10  to  12  aircraft. 

u.  F35Lead 

F-35  lead  distance  is  the  distance,  in  nautical  miles,  that  the  F-35C  JSF  is 
in  front  of  the  strike  aircraft.  This  lead  distance  is  important  in  that  it  gives  the  JSF  time 
to  detect  and  destroy  a  threat  before  the  threat  targets  the  strike  aircraft.  This  distance 
varies  from  0  to  200  nautical  miles.  F-35  lead  distance  is  converted  to  a  y-coordinate 
(y  =  27.5  to  47.5)  start  location  when  inputted  into  the  MANA  model. 

V.  F35Stlth 

F-35  stealth  is  the  aircraft  concealment  in  percent.  This  value  varies  from 
98  to  100.  These  values  correspond  to  a  RCS  from  0.005  to  0.0015  m^  (Grining,  2000).  A 
conversion  of  \Q0 -  {RCS y<\00Q)  =  concealment  is  used.  Integer  values  are  required  for 
MANA,  therefore  concealment  is  rounded  to  the  nearest  integer  value. 

w.  F35SnsrRng 

F-35  sensor  range  is  the  distance  at  which  the  F-35C  JSF  classifies  other 
squads.  This  value  varies  from  60  to  120  nautical  miles.  These  distances  are  based  on  the 
AN/APG-81  RADAR  characteristics  (Northrop  Grumman,  2010). 

X.  F35SmrApture 

F-35  sensor  aperture  is  the  sensor  arc  width.  This  value  varies  from  40  to 

180  degrees  (Northrop  Grumman,  2010).  Squads  that  are  within  the  range  and  aperture  of 
the  F-35  sensor  will  be  classified;  squads  that  are  outside  the  range  or  aperture  of  the 
F-35  sensor  will  not. 
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2. 


Uncontrollable  Factors 


The  following  uncontrollable  factors  were  chosen  for  the  experimental  design. 
For  consistency,  the  labels  are  the  same  as  those  in  MANA  output. 

a.  JlOCapQty 

J-10  CAP  quantity  is  the  number  of  J-10  aircraft  that  are  on  CAP  station. 
This  number  varies  from  1  to  20.  The  upper  limit  of  20  is  chosen  because  modern  day 
aircraft  are  rarely  deployed  in  groups  larger  than  12  to  15  aircraft. 

b.  JlOCapSpd 

J-10  CAP  speed  is  the  speed,  in  knots,  of  the  J-10  aircraft.  J-10  CAP  speed 
varies  from  250  to  800  knots  (Jane's  All  the  World's  Aircraft,  2010).  J-10  CAP  speed  is 
converted  to  MANA  units  by  dividing  the  speed  (in  knots)  by  ten. 

c.  JlOCapStlth 

J-10  CAP  stealth  is  the  aircraft  concealment  in  percent.  This  value  varies 
from  25  to  90.  Values  used  are  similar  to  the  F/A-18  Super  Hornet.  These  values 
correspond  to  a  RCS  of  0.075  to  0.01  m^  (Grining,  2000).  A  conversion  of 
1 00  -  {RCS  X 1 000)  =  concealment  is  used. 

d.  PLllAmmo 

PL- 12  weapons  quantity  is  the  number  of  PL- 12  active-RADAR  missiles 

carried  by  the  J-10.  PL-12  weapons  quantity  varies  from  1  to  6.  This  factor  is  lock- 
stepped,  therefore  every  CAP  J-10  and  Alert  J-10  carries  the  same  quantity  of  PL-12 
missiles. 
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e.  PL12WpnRng 


PL-12  weapons  range  is  the  range  of  the  PL-12  missile  earried  by  the  J-10. 
This  value  varies  from  35  to  50  nautieal  miles  (Jane's  Air-Launehed  Weapons,  2009). 
This  faetor  is  look-stepped  between  both  the  CAP  J-10  and  the  Alert  J-10. 

f.  PLUWpnPhit 

PL- 12  weapon  probability  of  hit  (Phit)  is  the  probability,  in  peroent,  that 
the  PL-12  missile  hits  the  target.  This  value  varies  from  0.0  to  1.00. 

g.  PLSAmmo 

PL-8  weapons  quantity  is  the  number  of  PL-8  infra-red  guided  missile 
earried  by  the  J-10.  PL-8  weapons  quantity  varies  from  2  to  4.  This  faetor  is  look-stepped, 
therefore  every  CAP  J-10  and  Alert  J-10  carries  the  same  quantity  of  PL-8  missiles. 

h.  PLSWpnRng 

PL-8  weapons  range  is  the  range  of  the  PL-8  missile  carried  by  the  J-10. 
This  value  varies  from  2  to  10  nautical  miles  (Jane's  Air-Launched  Weapons,  2009).  This 
factor  is  lock-stepped  between  both  the  CAP  J-10  and  the  Alert  J-10. 

L  PLSWpnPhit 

PL-8  weapon  probability  of  hit  (Phit)  is  the  probability  that  the  PL-8 
missile  hits  the  target.  This  value  varies  from  0.0  to  1.00. 

j.  JlOAlertQty 

J-10  alert  quantity  is  the  maximum  number  of  J-10  aircraft  that  are  in  alert 

status.  The  J-10  alert  aircraft  will  randomly  launch  from  the  enemy  airfield.  This  number 
varies  from  1  to  20.  The  upper  limit  of  20  is  chosen  because  modern  day  aircraft  are 
rarely  deployed  in  groups  larger  than  12  to  15  aircraft. 
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k.  JlOAlertSpd 


J-10  alert  speed  is  the  speed,  in  knots,  of  the  J-10  aireraft.  J-10  alert  speed 
varies  from  250  to  800  knots  (Jane's  All  the  World's  Aircraft,  2010).  J-10  alert  speed  is 
converted  to  MANA  units  by  dividing  the  speed  (in  knots)  by  ten. 

1.  JlOAlertStlth 

J-10  alert  stealth  is  the  aircraft  concealment  in  percent.  This  value  varies 
from  25  to  90.  Values  used  are  similar  to  that  of  the  F/A-18  Super  Hornet.  These  values 
correspond  to  a  RCS  of  0.075  to  0.01  m^  (Grining,  2000).  A  conversion  of 
1 00  -  {RCS  X 1 000)  =  concealment  is  used. 

C.  THE  EXPERIMENT 

We  built  the  MANA  model  in  steps.  The  first  step  involved  researching  and 
manually  entering  all  the  agents,  behaviors,  and  variables  for  the  baseline  model  into  the 
MANA  simulation.  MANA’s  well-documented  and  intuitive  menus  made  this  process 
fairly  easy,  but  this  was  still  the  most  time-consuming  step. 

Once  all  the  variables  were  entered  into  the  MANA  simulation,  the  process  of 
testing  the  model  began.  We  accomplished  this  by  running  the  model  while  adding  agents 
and  variables  in  steps.  First;  we  created  and  tested  the  interactions  between  friendly  and 
J-10  CAP  aircraft.  Second,  we  modeled  and  tested  the  interaction  between  the  NUCAS 
strike  aircraft  and  the  target  area.  Third,  we  added  the  J-10  alert  aircraft  and  coded  them 
to  randomly  launch  from  their  start  position.  Finally,  we  examined  many  individual 
model  runs  to  ensure  proper  interaction  among  all  agents. 

Testing  and  debugging  the  interactions  of  squads  was  essential  to  ensure  that  the 
model  simulated  real-world  operations  appropriately.  We  accomplished  this  by  the 
extensive  use  of  two  features  of  MANA — the  control  buttons,  and  the  squad  situational 
awareness  map. 
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The  control  button  functions  can  used  to  run,  pause,  reset,  and  slow  down  the 
model.  In  addition,  a  random  seed  value  can  be  entered  and  locked  in  order  to  investigate 
a  particular  behavior.  Figure  4  shows  a  screen  capture  of  the  control  functions. 


Eause 

»  Run 

Qf  MultiRun 

\<  Reset 

Step  Delay[ins] 


^  l-r) 

1 

M<ix.  Steps 

999  [:{ 

Seed 

LockO  1069687824 


Figure  4.  Screen  Capture  of  the  Various  Control  Buttons  (From  Defense 
Technology  Agency,  2010) 

We  used  the  squad  situational  awareness  map  to  troubleshoot  the  interactions  of 
various  squads.  A  wide  variety  of  information  on  situational  awareness  for  each  squad 
can  be  shown,  including  detections,  friendly  squad  classifications,  and  threat  squad 
classifications  with  their  associated  threat  level.  Figure  5  shows  the  situational  awareness 
map  for  the  F-35C  JSF  at  the  beginning  of  the  simulation.  The  red  squares  in  the  center  of 
the  screen  have  been  classified  as  enemies,  while  the  white  squares  are  contacts  that  have 
been  detected  but  are  not  within  the  range  to  be  classified.  The  blue  triangles  are  the 
friendly  units  that  are  in  the  same  squad. 
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Figure  5.  Screen  Capture  of  Squad  Situational  Awareness  Map  (From 
Defense  Technology  Agency,  2010) 

After  completing  the  MANA  simulation  and  ensuring  that  the  model  was  working 
correctly,  it  was  time  to  run  the  experiment.  To  do  this,  we  needed  an  efficient 
experimental  design.  Lieutenant  Colonel  Thomas  M.  Cioppa  developed  a  class  of 
experimental  designs  called  Nearly  Orthogonal  Latin  Hypercubes  (NOLHs)  while 
earning  his  doctorate  at  NFS,  Monterey,  CA.  The  NOTH  design  “combines  orthogonal 
Latin  hypercubes  and  uniform  designs  to  create  designs  having  near  orthogonality  and 
excellent  space-fdling  properties”  (Cioppa  and  Lucas,  2007).  Hernandez  et  al.  (2009) 
propose  variants  of  these  designs  that  are  even  more  efficient  and  allow  a  larger  number 
of  factors  to  be  explored.  In  this  thesis,  we  examine  36  different  factors  with  some  factors 
having  a  wide  variety  of  levels  between  their  minimum  and  maximum  values.  If  this 
thesis  were  to  just  evaluate  the  eight  NUCAS  factors  and  their  associated  levels,  the 
number  of  different  combinations  would  reach  1.145  trillion!  The  value  of  the  NOLH  is 
that  a  broad  section  of  the  input  space  can  be  covered  without  having  to  produce 
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simulations  on  the  entire  set  of  input  eombinations.  In  the  ease  of  this  thesis,  we  examine 
240  different  eombinations  by  implementing  the  NOHL  design,  using  the  ranges  of  factor 
levels  specified  in  Table  2  and  Table  4  for  the  uncontrollable  and  controllable  factors, 
respectively.  The  scatterplot  matrix  of  Figure  6  shows  the  orthogonality  and  space-filling 
properties  of  the  NOHL  design  used  for  the  24  controllable  factors.  The  portions  of  the 
design  for  the  uncontrollable  factors  (not  shown)  are  similar.  Although  some  factors  take 
on  a  limited  number  of  discrete  levels,  which  tends  to  degrade  the  orthogonality,  the 
maximum  pairwise  correlation  between  any  two  columns  is  less  than  0.1. 
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Figure  6.  Scatterplot  Matrix  of  Controllable  Factors  (From  SAS  Institute, 

2010) 
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These  240  eombinations  were  then  entered  into  an  exeeutable  seripting  file  ealled 
oldmedata,  written  by  my  assoeiate  Steve  Upton  of  NFS,  that  automatically  updates  the 
MANA  XML  file,  produces  a  separate  XML  file  for  each  of  the  240  different 
combinations,  and  launches  these  on  a  computing  cluster,  in  order  to  automate  the 
MANA  simulated  runs  and  then  collect  the  output  data  into  a  single  comma-separated 
(CSV)  file.  These  XML  files  are  then  ran  on  a  cluster  of  68  processors  operated  by  the 
Simulation  Experiments  and  Efficient  Design  (SEED)  Center  for  Data  Earming  at  NFS. 
50  runs  of  each  MANA  XML  file  were  executed  making  for  a  total  of  12,000  individual 
simulations.  On  a  single  processor,  12,000  runs  would  have  taken  50  hours.  Using  the  68- 
processor  cluster,  it  took  just  45  minutes,  emphasizing  the  advantage  of  using  NOLH 
experimental  design  in  combination  with  the  computational  power  of  the  cluster. 

The  entire  process  of  creating  the  data  file  from  the  MANA  simulation  was 
surprisingly  short,  especially  when  compared  to  the  extensive  amount  of  research  and 
development  that  went  into  creating  and  testing  the  baseline  MANA  simulation  model. 
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IV.  DATA  ANALYSIS 


A.  DATA  STRUCTURE 

The  single  CSV  output  file  eontains  the  following  additional  information: 

•  All  eontrollable  and  uneontrollable  faetors  with  their  appropriate  values 

•  Run  numbers 

•  Random  seed  numbers 

•  Steps  required  to  run  the  simulations 

•  Casualty  rates  for  every  squad 

•  Injury  rates  for  every  squad 

All  analysis  was  performed  on  this  CSV  file  exelusively  using  Version  8.0  of  a 
statistieal  paekage  developed  by  the  SAS  Institute  ealled  IMP  (SAS  Institute,  2010).  As 
we  state  in  Chapter  I,  two  Measures  of  Effeetiveness  (MOEs)  are  of  interest  in  this  thesis: 
Proportion  of  target  destruetion,  and  proportion  of  friendly  survivors.  Generating  the 
pereentage  of  friendly  easualties  was  aeeomplished  by  taking  the  number  of  friendly 
aireraft  remaining  at  the  end  of  the  simulation  and  dividing  by  the  number  of  friendly 
aireraft  available  at  the  start  of  the  simulation.  The  blue  easualty  proportion  was 
eonverted  to  blue  survivability  proportion  by  taking  1  minus  the  blue  easualty  proportion. 
A  conversion  from  meters  to  miles  was  also  performed  in  order  to  return  the  data  to 
United  States  customary  systems  units:  1  meter  is  equal  to  0.000621  miles. 

B,  INSIGHT  INTO  RESEARCH  QUESTIONS 

Chapter  I  lists  the  two  research  questions  that  directed  the  thesis: 

•  What  combination  of  manned  and  unmanned  aircraft  provides  the  best 
mission  success  rate? 

•  What  operational  factors  allow  the  NUCAS  to  complete  its  mission  with 
limited  loss  rate? 
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A  histogram  of  blue  survivability  shows  that  a  large  proportion  (1180 
simulations)  of  the  12,000  simulated  runs  result  in  zero  pereent  blue  survivability  rate. 
The  histogram  with  quantiles  and  moments  is  shown  in  Figure  7. 
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Figure  7.  Flistogram  of  Blue  Survivability  (From  SAS  Institute,  2010) 


A  histogram  of  target  casualty  shows  the  binomial  characteristics  of  target 
casualty  rate  being  either  destroyed  (1)  or  not  destroyed  (0).  The  mean  of  0.505  indicates 
that  there  are  slightly  more  destroyed  targets  (6064  simulations)  than  undestroyed  targets 
(5936  simulations). 

Throughout  the  rest  of  this  chapter,  the  summary  data  file  that  comes  from 
collapsing  over  the  50  replications  for  each  of  the  different  input  factor  combinations  is 
referenced.  This  summary  data  file  has  only  240  rows  of  data;  each  row  contains  the 
input  factor  settings  for  one  design  point,  along  with  the  mean  of  each  of  the  casualty 
statistics  and  MOEs  for  that  design  point.  This  summarization  has  some  advantages  for 
graphical  displays  and  comparisons,  and  does  not  affect  the  magnitudes  of  any  factor 
effect  estimates  in  regression  models  or  regression  trees.  However,  the  reader  should  note 
that  the  R  Squared  (R  )  for  the  regression  models  and  regression  trees  are  higher  when 
used  to  predict  the  mean  of  50  observations  than  when  used  on  the  raw  data.  Note  also 
that  some  of  graphs  and  charts  in  this  chapter  use  the  labels  “Mean(BlueSurv%)”  but 
these  are  really  proportions  (varying  from  zero  to  one)  rather  than  percents  (varying  from 
zero  to  100). 
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1. 


What  Combination  of  Aircraft  Provides  the  Best  Mission  Success 
Rate? 


Mission  success  is  a  function  of  the  target  easualty  proportion.  We  used  stepwise 
regression  to  fit  the  mean  target  easualty  proportion  using  all  eontrollable  input  factors, 
quadratic  effects,  and  all  two-way  interactions  as  potential  terms  in  the  model.  Figure  8 
shows  the  signilieant  terms  that  explain  roughly  88  pereent  of  the  variability  in  the  mean 
target  easualties.  By  eonvention,  main  effeets  are  retained  in  the  model  for  any  faetor 
appearing  with  signilieant  interaetion  or  quadratie  effeets. 
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Figure  8.  Regression  Analysis  of  Mean  Proportion  of  Target  Casualties 
(After  SAS  Institute,  2010) 
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The  significant  factors  that  have  the  most  influence  are  GBU-31  Phit,  NUCAS 
quantity,  weapon  quantity,  stealth,  and  sensor  aperture.  A  high  R  value  of  88  percent  is 
obtained  without  considering  the  uncontrollable  factors  in  the  regression.  This  suggests 
that  the  uncontrollable  factors,  like  enemy  quantity,  weapons  range,  and  stealth,  do  not 
play  as  nearly  an  important  role  on  mean  target  casualty  rate  as  the  controllable  factors. 


Regression  tree  analysis  is  another  useful  tool  to  determine  influential  factors,  and 
it  avoids  problems  of  predictions  outside  of  the  [0,1]  range  for  target  casualty  rates. 
Figure  9  shows  a  regression  tree  using  the  response  variable  mean  target  casualty  and  all 
the  controllable  input  factors.  This  regression  tree  has  five  splits  with  an  R  value  of 
0.641.  This  tree  is  read  from  top  to  bottom,  with  the  top  factors  having  a  larger  influence 
on  the  response  variable.  Branches  on  the  right  are  preferred  since  the  goal  is  to  have  a 
higher  target  casualty  percentage.  This  is  shown  in  Figure  9,  by  a  red  box  indicating  the 
poorest  performing  factor  combinations  and  a  green  box  indicating  the  best  performing 
factor  combinations. 
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Figure  9.  Regression  Tree  for  Mean  Target  Casualty  (After  SAS  Institute, 

2010) 


Many  of  the  same  factors  that  are  identified  as  important  with  the  stepwise 

regression,  are  also  important  using  the  regression  tree;  GBU-3 1  Phit,  NUCAS  quantity, 

and  sensor  aperture.  The  regression  tree  also  includes  information  about  NUCAS 
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quantity:  When  the  GBU-3 1  has  a  high  Phit,  then  missions  involving  quantities  of  less 
than  three  NUCAS  aireraft  have  an  average  mean  target  easualty  rate  of  only  0.22,  while 
missions  involving  three  or  more  NUCAS  have,  on  average,  a  mean  target  easualty  rate 
of  0.70.  This  indieates  that  three  or  more  NUCAS  is  desirable  beeause  substantial 
improvements  in  target  casualty  rates  can  be  achieved. 

Graphing  the  mean  target  casualty  rate  versus  NUCAS  quantity  gives  further 
insight  to  using  three  or  more  NUCAS  aircraft.  A  box  plot  of  the  mean  target  casualty 
rate  versus  NUCAS  quantity  (Figure  10)  shows  a  very  distinct  difference  of  the  slope  of 
mean  target  casualty  from  zero  to  three  NUCAS  aircraft.  The  slope  levels  out  after  three, 
suggesting  that  having  a  larger  number  of  NUCAS  aircraft  is  preferred  but  the  marginal 
gains  are  smaller  beyond  three  NUCAS  aircraft. 


Figure  10.  Box  Plot  of  Mean  Target  Casualty  Rate  Versus  NUCAS 
Quantity  (After  SAS  Institute,  2010) 


We  evaluate  NUCAS  weapon  quantity  and  stealth  further  by  using  box  plots  as 
shown  in  Figure  1 1 . 
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1.2 


Mean(TargetCas%)  vs.  UCASWpnQnty 


Mean(TargetCas%)  vs.  UCASStIth 


Figure  1 1 .  Box  Plots  of  Mean  Target  Casualty  Rate  Versus  NUCAS 

Weapons  Quantity  and  NUCAS  Stealth  (From  SAS  Institute, 
2010) 


Both  of  the  box  plots  of  target  easualty  rates  versus  NUCAS  weapons  quantity 
and  NUCAS  stealth  show  the  mean  values  are  elearly  improving  as  weapons  quantity  and 
stealth  improve.  A  military  signifieanee  ean  also  be  inferred,  espeeially  when 
improvements  in  mean  target  easualty  rate  inerease  from  approximately  40  to  60  percent. 

GBU-31  Phit  and  sensor  aperture  are  major  factors  determined  from  the 
regression,  and  deserves  further  exploration.  Using  the  prediction  profile  option  of  JMP, 
a  curve  of  the  expected  mean  target  casualty  percentage  versus  each  factor  is  shown. 
Figure  12  shows  the  prediction  profde  curves  for  both  GBU-31  Phit  and  NUCAS  sensor 
aperture.  The  polynomial  effects  on  the  regression  model  are  shown  in  the  curvature  of 
the  mean  target  casualty  line.  The  mean  target  casualty  percentage  versus  GBU-31  Phit 
displays  significant  gains  as  GBU-3 1  Phit  increases  but  begins  to  level  off  after  GBU-3 1 
Phit  of  0.8  percent.  Similarly,  the  mean  target  casualty  percentage  versus  NUCAS  sensor 
aperture  displays  significant  gains  as  the  sensor  aperture  increases,  but  gains  level  off  and 
then  actually  decrease  with  sensor  aperture  of  greater  than  92  degrees. 
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Figure  12.  Side-by-Side  Prediction  Profile  Curves  of  Mean  Target  Casualty 
Rate  Versus  GBU-31  Phit  and  NUCAS  Sensor  Aperture  (After 
SAS  Institute,  2010) 

In  conclusion,  the  success  of  the  target  destruction  is  a  direct  result  of  the  NUCAS 
factors.  This  conclusion  makes  sense  in  the  simulation;  the  NUCAS  is  the  only  aircraft 
capable  of  destroying  the  ground-based  target.  One  might  expect  that  manned  F-35C  and 
F/A-18  quantities  would  also  be  important  factors  since  the  survivability  of  the  NUCAS 
is  directly  related  to  how  well  the  manned  aircraft  protect  the  NUCAS.  Flowever,  based 
on  the  regression  analysis,  F-35C  quantity  and  F/A-18  quantity  are  not  major 
determinants  of  the  target  casualty  rate.  This  may  suggest  that  NUCAS  aircraft  have 
relative  success  if  used  autonomously,  without  OCA  support  of  the  F-35C  or  F/A-18. 

2,  What  Factors  Contribute  to  Minimal  Blue  Loss  Rate? 

Mission  success  is  determined  by  effectively  destroying  the  target.  However, 
losing  a  majority  of  the  aircraft  directly  affects  the  CAW’s  ability  for  further  strikes. 
Developing  tactics  that  ensure  target  destruction  while  minimizing  blue  loss  rate  is  of 
vital  importance  if  future  TACAIR  operations  are  necessary. 

A  stepwise  regression  on  mean  blue  survivability  rate  was  performed  using  all  the 
controllable  input  factors,  quadratic  terms,  and  two-way  interactions.  Figure  13  shows  the 
significant  factors  that  explain  roughly  60  percent  of  the  variability  in  the  response. 
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Figure  13 
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Regression  Analysis  of  Mean  Blue  Survivability  Proportion 
(After  SAS  Institute,  2010) 
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The  significant  factors  that  have  the  most  influence  are  F-35C  quantity,  NUCAS 
quantity,  AIM-120  Weapon  Phit,  and  stealth  of  the  NUCAS  and  F-35C. 

Figure  14  shows  a  regression  tree  of  mean  blue  survivability  using  all  the 
controllable  input  factors  as  potential  explanatory  factors.  This  regression  tree  has  five 
splits  with  an  R  value  of  0.275.  Like  the  previous  tree,  this  tree  is  read  from  top  to 
bottom,  with  the  top  factors  having  a  larger  influence  on  the  response  variable.  Branches 
on  the  right  tend  to  be  preferred  since  the  goal  is  to  have  a  higher  target  casualty 
percentage.  This  is  shown  in  Figure  14  by  a  red  box  indicating  the  poorest  performing 
factor  combinations,  and  a  green  box  indicating  the  best  performing  factor  combinations. 


Figure  14.  Regression  Tree  for  Mean  Blue  Survivability  Percentage  (After 
SAS  Institute,  2010) 


The  regression  tree  shows  that  many  of  the  same  factors  that  are  important  with 
the  stepwise  regression  are  also  important  using  the  regression  tree:  F-35C  quantity, 
NUCAS  quantity,  F-35C  stealth,  and  NUCAS  stealth.  Quantities  of  less  than  nine  F-35C 
aircraft  have  the  greatest  effect  on  blue  survivability  percentage.  The  regression  tree  also 
highlights  that,  even  with  the  small  number  of  splits,  there  is  a  wide  variation  in  mean 
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blue  survivability  rates,  ranging  from  0.33  to  0.81  (33  pereent  to  81  pereent).  This 
suggests  that  F-35C  aireraft  quantities  greater  than  nine  are  not  enough  to  lead  to  high 
mean  blue  survivability  rates. 

For  a  more  detailed  look  at  the  effects  of  F-35C  quantity  on  survivability,  a  box 
plot  of  the  mean  blue  survivability  rate  versus  F-35C  quantity  is  shown  in  Figure  15.  A 
similar  box  plot  showing  mean  blue  survivability  rate  versus  NUCAS  quantity  is  shown 
in  Figure  16. 


Figure  15.  Box  Plot  of  Mean  Blue  Survivability  Percentage  Versus  F-35C 

Quantity  (After  SAS  Institute,  2010) 

It  is  of  interest  to  note  that  when  a  line  is  drawn  through  the  mean  of  each  of  the 
F-35C  quantities,  the  curve  is  noticeably  steeper  from  three  to  four  (and  again  from  eight 
to  ten  aircraft).  This  suggests,  that  based  on  the  simulation  data,  increasing  the  quantity  of 
F-35Cs  aircraft  from  four  to  eight  does  not  offer  considerable  advantages.  However, 
using  nine  or  more  aircraft  does  improve  the  blue  survivability  percentage. 


45 


1 


Mean(BlueSurv%)  vs.  UCASQnty 


Legend 


Figure  16.  Box  Plot  of  Mean  Blue  Survivability  Pereentage  Versus 
NUCAS  Quantity  (After  SAS  Institute,  2010) 


When  evaluating  the  box  plot  of  mean  blue  survivability  pereentage  versus 
NUCAS  quantity,  a  similar  signifieant  reduetion  in  mean  blue  survivability  pereentage 
oeeurs  when  NUCAS  quantity  is  greater  than  three  aireraft.  This  is  espeeially  interesting 
since  three  or  more  NUCAS  aircraft  is  the  preferred  number  when  mean  target  casualty 
rate  is  the  response  variable.  Figure  17  is  a  side-by-side  comparison  of  mean  target 
casualty  rate  and  mean  blue  survivability  rate  versus  NUCAS  quantity: 


Figure  17.  Side-by-Side  Comparison  of  Mean  Target  Casualty  Rate  and 
Mean  Blue  Survivability  Rate  Versus  NUCAS  Quantity  (After 
SAS  Institute,  2010) 
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NUCAS  and  F-35  stealth  are  major  factors  determined  from  the  regression,  and 
deserve  further  exploration.  Using  the  prediction  profile  option  of  JMP,  a  curve  of  the 
expected  mean  blue  survivability  percentage  versus  each  factor  is  shown.  Figure  18 
shows  the  prediction  profile  curves  for  both  NUCAS  stealth  and  F-35  stealth.  Both  of  the 
curves  clearly  show  that  blue  survivability  rate  increases  with  stealth.  Both  of  these 
graphs  show  the  military  significance  of  stealth  in  mean  blue  survivability.  Small 
percentile  gains  of  just  0.02  in  stealth  result  in  gains  of  at  least  15  percent  in  mean  blue 
survivability. 


Figure  18.  Prediction  Profile  Curves  of  Mean  Blue  Survivability 

Percentage  Versus  NUCAS  Stealth  and  F-35C  Stealth  (After 
SAS  Institute,  2010) 


AIM-120  weapon  Phit  is  another  major  factor  determined  from  the  regression,  and 
deserves  further  exploration.  A  prediction  profile  curve  showing  blue  survivability  rate 
versus  AIM- 120  Weapon  Phit  is  shown  in  Figure  19.  This  linear  relationship  shows  a 
significant  increase  in  mean  blue  survivability  rate  as  AIM-120  Phit  increases.  This 
highlights  the  importance  of  continued  development  of  weapons  designed  to  increase  the 
Phit  characteristics. 
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Figure  19.  Prediction  Profile  Curve  of  Mean  Blue  Survivability  Percent 
Versus  AIM-120  Weapons  Phit  (After  SAS  Institute,  2010) 

In  conclusion,  the  major  factors  that  reduce  the  blue  survivability  percentage 
include  F-35C  quantity,  NUCAS  quantity,  AIM-120  weapon  Phit,  NUCAS  stealth,  and  F- 
35C  stealth.  F-35C  quantity  is  the  most  important  factor;  however,  no  substantial  increase 
in  blue  survivability  rates  are  seen  when  the  quantity  of  F-35C  aircraft  ranged  from  four 
to  nine  aircraft.  NUCAS  quantity  is  the  second  most  important  factor,  showing  that  three 
or  more  NUCAS  aircraft  are  preferred. 


48 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  RESEARCH  QUESTIONS 

This  purpose  of  this  thesis  is  to  gain  some  insight  to  the  following  questions: 
What  eombination  of  manned  and  unmanned  aireraft  provides  the  best  mission  suceess 
rate?  What  operational  faetors  allow  the  NUCAS  to  eomplete  its  mission  with  limited 
loss  rate  to  manned  and  unmanned  aireraft? 

1.  What  Combination  of  Aircraft  Provides  the  Best  Mission  Success 
Rate? 

When  the  response  faetor  was  target  easualty  pereentage,  the  regression  model 
analysis  shows  that  NUCAS  quantities  of  three  or  more  produee  signifieant  gains  in 
mission  sueeess  rate.  Box  plot  analysis  shows  that  maximizing  the  NUCAS  weapons 
quantity  and  stealth  are  militarily  signifieant  faetors.  Using  the  regression  model 
produeed  from  stepwise  regression,  the  predietion  profder  eurves  show  that  NUCAS 
sensor  apertures  near  92  degrees  lead  to  improved  target  easualty  rates.  The  predietion 
profile  eurves  also  show  that  improvements  to  GBU-3 1  weapon  Phit  pereentages  lead  to 
inereases  in  target  easualty  rates  as  well. 

2,  What  Factors  Contribute  to  Minimal  Blue  Loss  Rate? 

When  blue  survivability  is  the  response  faetor,  regression  analysis  shows  that  a 
NUCAS  quantity  of  three  or  more  aireraft  produees  signifieant  gains  in  blue 
survivability.  F-35C  quantities  between  four  and  nine  aireraft  also  aid  in  blue 
survivability  rates.  The  predietion  profiler  eurves  show  that  NUCAS  and  F-35C  stealth 
are  important  faetors.  In  addition,  the  predietion  profile  eurves  shows  AIM- 120  weapon 
Phit  is  another  important  faetor  in  the  survivability  of  blue  aireraft. 
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B,  RECOMMENDATIONS 

We  recommended  that,  based  on  the  scenario  and  simulation  data  provided: 

•  NUCAS  aircraft  should  be  operated  in  a  group  of  at  least  three  to  four 
aircraft.  Military  aviators  have  a  long  history  of  working  in  a  section  of 
two  or  a  division  of  four  aircraft.  Based  on  the  reality  of  equipment 
failure,  we  recommended  that  NUCAS  operate  in  a  division  of  four 
aircraft  in  order  maximize  both  probability  of  target  destruction  and  blue 
survivability. 

•  F-35C  aircraft  should  operate  in  a  group  of  at  least  four  aircraft  when  used 
for  OCA  missions  against  a  large  number  of  enemy  aircraft.  Marginal 
gains  in  blue  survivability  are  achieved  by  increasing  the  quantity  of  F- 
35Cs  beyond  four  aircraft. 

•  Stealth  is  an  important  factor  for  both  successful  target  destruction  and 
blue  survivability.  The  F-35C  proved  to  outperform  the  F/A-18  even  when 
sensor  and  weapons  parameters  were  equal  among  the  two  aircraft. 

•  Weapons  Phit  is  a  major  factor  in  target  destruction  and  blue  survivability. 
Weapons  technology  advances  must  continue  to  improve  thus  ensuring  Phit 
efficiency  is  maintained. 

C.  CONTINUED  RESEARCH 

This  thesis  originally  included  three  scenarios  that  were  to  be  evaluated.  In 
addition  to  the  coordinated  strike  scenario,  a  Defensive  Counter  Air  (DCA)  scenario  and 
an  autonomous  NUCAS  long-range  strike  were  developed.  Both  of  the  scenarios  were 
implemented  into  MANA  with  extensive  testing  done  on  both  simulation  models. 

The  DCA  scenario  model  uses  F/A-18  and  F-35C  manned  aircraft  that  are 
positioned  at  a  Combat  Air  Patrol  (CAP)  station.  The  CAP  aircraft  intercept  and  destroy 
inbound  strike  aircraft  that  approach  randomly.  The  NUCAS  is  equipped  as  an  in-flight 
refueling  platform  with  external  tanks  and  a  central  refueling  pod.  As  F/A-18  and  F-35C 
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aircraft  require  fuel,  the  NUCAS  will  intereept  and  rendezvous  with  manned  aireraft  to 
provide  fuel  necessary  for  eontinued  airborne  operations.  Researeh  questions  inelude; 
How  many  NUCAS  and  DCA  aireraft  are  needed  in  order  to  provide  statistical  gains  in 
DCA  effectiveness?  Which  factors  are  most  important  in  managing  DCA  effectiveness? 
Figure  20  shows  a  MANA  screen-shot  of  the  DCA  scenario. 


Figure  20.  MANA  Sereen  Shot  of  DCA  Seenario  (From  Defense 
Teehnology  Ageney,  2010) 

The  autonomous  long-range  strike  seenario  model  was  developed  to  look  strietly 
at  the  factors  of  the  NUCAS.  The  scenario  is  a  long-range  strike  where  NUCAS  aircraft 
are  designated  to  destroy  a  target  deep  into  enemy  territory.  Threats  include  long-range 
strategic  and  medium  range  tactical  Surface  to  Air  Missile  (SAM)  sites.  The  NUCAS 
route  is  predetermined  to  fly  into  the  enemy  territory,  strike  the  target,  and  then  egress 
while  avoiding  the  Weapon  Engagement  Zones  (WEZ)  of  the  strategic  SAM  threats.  The 
taetical  SAM  threats  are  randomized  throughout  the  enemy  territory.  If  the  NUCAS 
successfully  detects  the  tactical  SAM  threat  then  the  UCAS  will  attempt  to  maneuver 

away  from  the  WEZ.  Researeh  questions  include;  How  many  UCAS  aireraft  are  required 
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to  successfully  destroy  the  target?  What  factors  are  crucial  to  the  suceess  of  the  mission? 
What  faetors  are  crucial  to  the  survivability  of  the  UCAS?  Figure  21  shows  a  MANA 
screen-shot  of  the  long-range  strike  scenario: 


Figure  21 .  MANA  Sereen  Shot  of  Long-Range  Strike  Scenario  (From 
Defense  Technology  Agency,  2010) 

Since  much  of  the  researeh  and  development  has  been  done  on  these  two 
scenarios,  it  would  be  advantageous  to  continue  with  the  design  of  experiments  and 
analysis  of  these  two  models.  This  information,  used  in  combination  with  the  coordinated 
strike  scenario  described  in  this  thesis,  would  provide  valuable  insight  into  the  future  of 
NUCAS  operations  and  CAW  combat  performance. 
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